UNIQUENESS OF BESSEL MODELS: THE ARCHIMEDEAN CASE 
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Abstract. In the archimedean case, we prove uniqueness of Bessel models for 
general linear groups, unitary groups and orthogonal groups. 



1. Introduction 
Let G be one of the classical Lie groups 

(1) GL n (R), GL n (C), U(p, q), 0(p, q), O n (C). 

In order to consider Bessel models for G, we consider, for each non-negative integer 
r satisfying 

n > 2r + 1, p > r, q > r + 1, 

the r-th Bessel subgroup 

S r = N Sr X Go 

of G, which is a semidirect product and which will be described explicitly in Section 
12.11 Here N Sr is the unipotent radical of S r , and G is respectively identified with 

(2) GL^a-iOR), GL^2r-i(C), V{p-r,q-r-l), 0{p-r,q-r -1), O n _ 2r _!(C). 

Let xs r be a generic character of S r as defined in Section 12. 2[ The main result of 
this paper is the following theorem, which is usually called the (archimedean) local 
uniqueness of Bessel models for G. 

Theorem A. Let G, Go, S r andxs r be as above. For every irreducible representation 
n of G and n of G both in the class TH., the inequality 

dimHom Sr (7r®7ro,X5 r ) < 1 

holds. 

We would like to make the following remarks on Theorem \M The symbol "(g)" 
stands for the completed projective tensor product of complete, locally convex topo- 
logical vector spaces, and "Homs r " stands for the space of continuous ^-intertwining 
maps. Note that ttq is viewed as a representation of S r with the trivial iVs r -action. 
As is quite common, we do not distinguish a representation with its underlying space. 
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Recall that a representation of G is said to be in the class FTi if it is Frechet, 
smooth, of moderate growth, admissible and Z(gc)-fi n he. Here and as usual, Z(g c ) 
is the center of the universal enveloping algebra U(gc) of the complexified Lie algebra 
0c of G. Of course, the notion of representations in the class J-'TC fits all real reductive 
groups. The interested reader may consult [Casj and |Wal| Chapter 11] for more 
details. 

If r = 0, then S r = Go, and Theorem |A] is the multiplicity one theorem proved by 
Sun and Zhu in |SZ] (and independently by Aizenbud and Gourevitch in [AG] for 
general linear groups). If Go is the trivial group, then Theorem lAl asserts uniqueness 
of Whitta ker mo dels for GL 2r+ i(M), GL 2r +i(C), U(r,r + 1), 0(r,r + l) and 2r +i(C). 
See |Shl] . |CHM] for local uniqueness of Whittaker models for quasi-split groups (or 
[JSZJ for a quick proof). Hence the family of Bessel models interpolates between the 
Whittaker model (Go is trivial) and the spherical model (r = 0). 

It is a basic problem in representation theory to establish various models with 
good properties. In particular, this has important applications to the classification 
of representations and to the theory of automorphic representations. 

Whittaker models for representations of quasi-split reductive groups over com- 
plex, real and p-adic fields and their local uniqueness property are essential to the 
Langlands-Shahidi method ( |Shh] ) and the Rankin- Selberg method ([Bum]) to es- 
tablish the Langlands conjecture on analytic properties of automorphic L-functions 

([SB]). 

The notion of Bessel models originates from classical Bessel functions and it was 
first introduced by Novodvorski and Piatetski-Shapiro QNPSJ) to study automorphic 
L-functions for Sp(4). For orthogonal groups, the Bessel models are essential to 
establish analytic properties of automorphic L-functions as considered in [GPSRJ. 
The analogue for unitary groups is expected (see |BAS] . for example). More recently, 
Bessel models are used in the construction of automorphic descents from the general 
linear groups to certain classical groups QGRSJ), as well as in the construction of local 
descents for supercuspidal representations of p-adic groups ([JS], [Sou] . |JNQ08| , and 
JNQ09|). Further applications of Bessel models to the theory of automorphic forms 
and automorphic L-functions are expected. 

We remark that the local uniqueness of the Bessel models is one of the key prop- 
erties, which makes applications of these models possible. An important purpose of 
this paper is to show that the archimedean local uniqueness of general Bessel models 
can be reduced to the uniqueness of the spherical models proved in |SZ] (i.e. r = 
case). The key idea in this reduction is to construct an integral L^ (Equation (fl"3l 
in Section [373]) . where \i is a (non-zero) Bessel functional. We note that for p-adic 
fields, the reduction to the p-adic spherical models (proved in [AGRS] ) is known by 
the work of Gan, Gross and Prasad ( |GGPj ). The approach of this paper works for 
the p-adic local fields as well. 
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We now describe the contents and the organization of this paper. In Section [2J we 
recall the general set-up of the Bessel models. In Section [3j we outline our strategy, 
and give the proof of Theorem A, based on two propositions on the aforementioned 
integral J M (Propositions 13. 31 and 13.41) . This integral depends on a complex parameter 
s. Proposition 13.31 states that J^, when evaluated at a certain point of the domain, 
is absolutely convergent and nonzero. On the other hand, Proposition 13.41 asserts 
that J M converges absolutely for all points of the domain when the real part of the 
parameter s is large, and it defines a G-invariant continuous linear functional on a 
representation of G' x G in the class FH, where G' D G is one of the spherical pairs 
considered in |SZ] . The proof of Proposition 13.31 and Proposition 13.41 are given in 
Sections H] and [61 respectively. Section [5] is devoted to an explicit integral formula 
(Proposition 15 .4[) , ClS db preparation for Section [6j 

Acknowledgements: the authors would like to thank D. Barbasch, J. Cogdell, D. 
Soudry, R. Stanton, and D. Vogan for helpful conversation and communication. Di- 
hua Jiang is supported in part by NSF grant DMS-0653742 and by the Chinese Acad- 
emy of Sciences. Binyong Sun is supported by NSFC grants 10801126 and 10931006. 
Chen-Bo Zhu is supported in part by NUS-MOE grant R-146-000-102-112. 

2. Bessel subgroups and generic characters 

2.1. Bessel subgroups. In order to describe the Bessel subgroups uniformly in all 
five cases, we introduce the following notations. Let K be a IR-algebra, equipped 
with an involution r. In this article, (K, r) is assumed to be one of the pairs 

(3) (lxR,r E ), (CxC,r c ), (C,-), (R,1 H ), (C,l c ), 

where tr and tq are the maps which interchange the coordinates, " — " is the complex 
conjugation, 1^ and lc are the identity maps. 

Let £ be a hermitian K-module, namely it is a free K-module of finite rank, 
equipped with a non-degenerate M-bilinear map 

( , ) E : E x E -> K 

satisfying 

(u, v)e — (v, u) T E , (au, v )e — a(u, v)e, a <E K, u,v e E. 

Denote by G := U(E) the group of all K-module automorphisms of E which preserve 
the form ( , ) E - 

Assume that E is nonzero. Let r > and 

= Xq C X\ C • ■ ■ C X r C X r +i 

be a flag of E such that 
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• Xi is a free K-submodule of E of rank z, z = 0, 1, • • • , r, r + 1, 

• X r is totally isotropic, and 

• X r+ i = X r © Kt> (orthogonal direct sum), with v' a non- isotropic vector. 
A group of the form 

(4) S r := {x E G | (x - l)X i+l C X h i = 0, 1, • • • , r} 
is called a r-th Bessel subgroup of G. 

To be more explicit, we fix a totally isotropic free K-submodule Y r of 

v' L := {vEE \ (v,v' ) E = 0} 
of rank r so that the pairing 

( , ) E : X r x F r -> K 

is non-degenerate. Write 

E := ^ n (X r © F r ) ± . 
Then is decomposed into an orthogonal sum of three submodules: 

(5) E = (X r © K r ) © E © Kuq. 

According to the five cases of (IK, r) in Q, G is one of the groups in (TTJ. By 
scaling the form ( , )e, we assume that 

Wq,Vq)e = -1, 

then G := U(-Eo) is one of the groups in The Bessel subgroup S r is then a 
semidirect product 

(6) S r = N Sr x G , 
where N Sr is the unipotent radical of S r . 

2.2. Generic characters. Write 

U := Rom K (X i+1 /Xi,Xi/Xi^i), i = 1,2, • • • ,r, 

which is a free IK- module of rank 1. For any x £ S r , x — 1 obviously induces an 
element of Lj, which is denoted by [sc — Denote by [x} the projection of x to G . 
It is elementary to check that the map 

r] r : S r — > G r := Go x Li x L 2 x ■ ■ ■ x L r , 

a; i-> ([x] , [a; - l]i, [x - 1] 2 , ■ • • , [x - l] r ) 

is a surjective homomorphism, and every character on S r descends to one on C r . A 
character on S r is said to be generic if its descent to C r has nontrivial restriction to 
every nonzero K-submodule of Lj, i = 1, 2, • • • , r. 
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3. The strategy, and proof of Theorem \M 
3.1. The group G'. Introduce 

E' :=E® Kv', 

with v' a free generator. View it as a hermitian K-module under the form ( , )e> so 
that 

(,)e'\exe = (,)e, (E,v') e < = and (v',v') E > = l. 
Then E' is the orthogonal sum of two submodules: 

E' = (X' r+1 ®Y; +1 )®E , 

where 

X' r+1 :=X r @ K(v' + v') and F r ' +1 := Y r © K(v' - v') 

are totally isotropic submodules. 

Write G' := XJ(E'), which contains G as the subgroup fixing v' . Denote by P r ' +1 
the parabolic subgroup of G' preserving X' r+1 , and by P r the parabolic subgroup of 
G preserving X r . As usual, we have 

(8) P r ' +1 = N P , +i x (Go x GL r+ i) C G' and 

(9) P r = Np r x (G' x GL, r ) c G, 

where N P ^ i and A?"p r are the unipotent radicals of P r ' +1 and P r , respectively, 
GL r+ i := GLk(X^, +1 ) D GL r := GLjK(X r ), 

and 

G' Q := U(E' ) D G , with E' := P © K^. 

Write 

N r+1 = {xe GL r+ i | (z - 1)X; +1 C X r , (x - l)Xi C i = 1, 2, • • • , r}, 

and 

N r = {x E GL r | (x - 1)X C Xi-i, i = 1, 2, • • • , r}, 
which are maximal unipotent subgroups of GL r+1 and GL r , respectively. 

We now describe other salient features of the Bessel group S r . It is a subgroup of 
P r : 

(10) S r = N Pr X (G x N r ) C P r = N Pr x (G' x GL r ). 

Although P r is not a subgroup of we have that 5 r C P T ' +1 and the quotient 

map P r ' +1 — > Go x GL r+ i induces a surjective homomorphism 

(11) fj r : S r ^ G X N r+ i. 
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It is elementary to check that every character on S r descends to one on G x N r+ \, 
and it is generic if and only if its descent to Go x N r+ i has generic restriction to 
N r+ i, in the usual sense. 

Let xs r be a generic character of S r , as in Theorem El Write 

(12) Xs r = (Xg ® Vv+i) °Vr, 

where xg is a character on Go, and ip r +i is a generic character on N r+ \. Throughout 
this article, we always assume that Vv+i is unitary. Otherwise the Horn space in 
Theorem |X] is trivial, due to the moderate growth condition on the representation ir. 

3.2. Induced representations of G' . Let 7r and a be irreducible representations 
of Go and GL r+1 in the class J-'Ti, respectively. Write 

p := 7r <S>cr, 

which is a representation of Go x GL r+ i in the class J-'Ti. 
Put 



and 

Denote by 



1, if IK is a field, 

2, otherwise, 



lX 



if d K = 1, 



L£ x M^, otherwise. 



|-| : K x -» Kl 

the map of taking componentwise absolute values. For all a G and s G C dm , put 

a s := a^ 1 a S 2 G C x , if d K = 2, a = (oi, a 2 ), s = (sj, s 2 ). 
If c?k = 1, a s G C x retains the usual meaning. 

We now define certain representations of G' in the class FTi which are induced 
from the parabolic subgroup -P r '+i- For every s G C* 4 , denote by ir' s the space of all 
smooth functions / : G' — > p such that 

f(n'gmx) = Xa^y 1 |det(m)| s p(gm)(f(x)), 

for all n' G N P ^ , G G , m G GL r+1 , x G G' . (We introduce the factor XGo(d) 1 
for convenience only.) 

By using Langlands classification and the result of Speh-Vogan |SVl Theorem 1.1], 
we have 

Proposition 3.1. The representation tt' s is irreducible except for a measure zero set 
ofse C dK . 
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3.3. The integral 1^. Recall that Vv+i is the generic unitary character of N r+ i as 
in (|T2l) . Assume that the representation a of GL r+1 is -generic, namely there 
exists a nonzero continuous linear functional 

A : a -> C 

such that 

X(a(m)u) = -?/Y + i(m) _1 A(w), m G N r+ \, u G cr. 
We fix one such A. Define a continuous linear map A by the formula 

A : p = ir ®(7 — > 7r 

u ® v i — > A(i>)«. 

Let 7r be an irreducible representation of G in the class JFK, as in Theorem and 

let 

jj : 7T X 7T — ► C 

be a Bessel functional, namely a continuous bilinear map which corresponds to an 
element of 

Rom Sr (iT®7io,xs r )- 
Lemma 3.2. For every s G C dK , u G 7r ane? f £ tt' s , the smooth function 

g i-> ^(tt(p)«, A(/(a))) 
on G is left invariant under S r . 
Proof Let x G G and b e S r C P r ' +1 . Write 

6 = n'gm, n' G Np^ +1 , g G Go, m G iV r+1 . 

Then 

A(/(te))) = XGoO/)- l A(pO/m)(/(a;))) 

= X Go (o)-Vr + i(m)- 1 7r (o)(A(/(a;))) 
= X5 r (&)^vro(o)(A(/(x))), 

and therefore, 

A(/(te))) = X5r (6)-V(7r(6)7r(x) W) 7r Q (o)(A(/(x)))) 
= (i(ir(x)u, A(/(x))). 



The last equality holds as 6 maps to o under the quotient map S r -» G , and 7T is 
viewed as a representation of 5 r via inflation. □ 
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Write 

(13) I p (f,u): = n(ir(g)u, A(f(g)))dg, f G ir' s , u G 7T, 

JSr\G 

where dg is a right G-invariant positive measure on S r \G. It is clear that 

I»«(g)f,n(g)u) = I,(f,u) 
for all g G G whenever the integrals converge absolutely. 

3.4. Proof of Theorem [A]. We shall postpone the proof of the following proposition 
to Section HI 

Proposition 3.3. If // ^ 0, then there is an element f p G n' s and a vector u n G 7r 
such that the integral I ^(fp,^) converges absolutely, and yields a nonzero number. 

Denote by 

Re : C dK -> R* 

the map of taking real parts componentwise. If a G M. dK and c e R, by writing a > c, 
we mean that each component of a is > c. 

The proof of the following proposition will be given in Section [6] after preparation 
in Section [51 

Proposition 3.4. There is a real number c M such that for all s G C dK with Re(s) > 
c p , the integral I p (f,u) converges absolutely for all f G 7r' s and all u G it, and 1^ 
defines a continuous linear functional on tt' s ®-k. 

We now complete the proof of Theorem [K\ We are given tt, tcq and a generic 
character xs r °f $r as i n Equation f|T2|) . As noted there, we may assume that the 
generic character ip r +i of iV r+ i is unitary. As is well known, there exists an irre- 
ducible representation a of GL r+1 in the class J-"H which is ipr+i -generic (it follows 
from jCHMt Theorem 9.1], for example). For each \i G Homs r (7r®7ro, Xs r )i we may 
therefore define the integral 1^, as in Section [3731 

Let F r be a finite dimensional subspace of Homs r (7r®7To, Xs r )- By Proposition 
I3.4[ there exists a real number cp r such that for all // G F r and all s G C dK with 
Re(s) > cf t , the integral I p (f, u) converges absolutely for all / G n' s and all u G n, 
and defines a continuous linear functional on n' s ^ir. 

By Proposition 13. 1\ we may choose one s with Re(s) > cp r and 7r^ irreducible. By 
Proposition 13.31 we have a linear embedding 

F r ^ Hom G (7r^®7r, C), // i-> J M . 

The later space is at most one dimensional by [SZ\ Theorem A], and so is F r . This 
proves Theorem IA1 □ 
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4. Proof of Proposition 13.31 

We continue with the notation of the last section and assume that fi ^ 0. By 
absorbing the concerned characters into the representations tt and a, we may and 
we will assume in this section that xg = 1 an d s = G C dK . 

Let Np r be the unipotent subgroup of G which is normalized by G' x GL r . so that 
(G' x GL. r )Np r is a parabolic subgroup opposite to P r . Then 

P r Np r is open in G. 

Recall that the Bessel group S r is a subgroup of P r : 

S r = N Pr x (G x JV r ) c P r = N Pr x (G' x GL r ). 

We shall need to integrate over S r \G, and thus over the following product space 

N r \GL r x (C7 \G ) x N Pr . 

4.1. A nonvanishing lemma on Go\G' . 

Lemma 4.1. There is a vector u n G 7r and a smooth function / wo : G' Q — > ttq, 
compactly supported modulo Gq such that 

Uo(gg') = no(9)U (g'), g e Go, g' e G , 

(14) / ^/^Dil^O. 

•/Go\<3' 

Proof. Pick u n E 7i and m,^ G 7r so that 

^{u % ,u m ) = 1. 

Let A' be a submanifold of G such that the multiplication map Go x A' ^ C7 is an 
open embedding, and 

Re(^(7r(a)w^,u 7ro )) > 0, a G A'. 

Let 0o be a compactly supported nonnegative and nonzero smooth function on A'. 
Put 

t ( n '\ ._ J 0o(a)7Tci(#)«7ro, if g' = gae G A', 
MAg ) ■ | o, otherwise, 

which clearly fulfills all the desired requirements. 

□ 
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4.2. Whittaker functions on GL r . Fix and as in Lemma [4. 11 Set 



$(m,n):= / n(7r {mg'n)u n , f* (g')) dg', 
Jg \g' 

which is a smooth function on GL r x Np r . It is nonzero since $(1,1) ^ by ( fT4|) . 
Note that /i is X^-equivariant , the representation ttq of S r has trivial restriction to 
N r , and xs r and ipr+i have the same restriction to N r . Therefore, we have 

$(Zm, n) = ip T+ i{l)<&{m, n), I G iV r , m G GL r , n G N Pr . 

Let W r be a smooth function on GL r with compact support modulo N r such that 

(15) W r (lm) = ip r+ i(l)- l W r (m), I G N r , m G GL r . 

The following lemma is due to Jacquet and Shalika ( [JaShl Section 3], see also 



Cog[ Section 4]). 



Lemma 4.2. For every W r as above, there is a vector u a G o such that 

W r (m) = X(a(m)u a ), m G GL r . 

Let (pfj be a smooth function on N Pr with compact support. Pick W r and (f>^ 
appropriately so that 

(16) / 5p r 1 (m)$(m, n)W r (m)(fif?(fi) dmdn ^ 0. 

J(N r \GL r )xNp r 

Here and as usual, we denote by 

S H : /ih |det(Ad ft )| 
the modular character of a Lie group H. 

4.3. The construction of f p . Note that 

(17) P' r+1 n G = N Pr x (G x GL r ). 

By counting the dimensions of the concerned Lie groups, we check that the multipli- 
cation map 

(18) P'r+i x G — > G' is a submersion. 
From (IT7I) and ( |T8l) . we see that the multiplication map 

t G , : (N K+i x GL r+1 ) x (G' x iVpJ -> G' 

is an open embedding. 
Put 

Mn)f*o(g > ) ® (o-("i)m ct ), if x = L G ,(n',m,g',n), 

0, if x is not in the image of 



/ P 0) : = 



UNIQUENESS OF BESSEL MODELS: THE ARCHIMEDEAN CASE 11 

where u a is as in Lemma 14.21 Then f p £ ir' s (recall that s is assumed to be 0) . 
Finally, we have that 

^(x)^, A(f p (x))) dx 

S r \G 

/x(7r(m^'n)u T , A(f p (mg'n)))Sp^(m) dmdg' dn 

(N r \GL r )x{G \G' )xNp r 

8 P }(m)X(a(m)u a )(l) R (n) 

(N r \GL r )x(G \G' )xNp r 

■ n{-K{mg'n)uir, U {g')) dmdg' dn 
5p 1 (m)$(m, n)W r (m)(f) £;(n)dm dn, 

(N r \GL r )xNp r 

which converges to a nonzero number by ([TBI) . This finishes the proof of Proposition 

5. Another integral formula on S r \G 

This section and the next section are devoted to a proof of Proposition 13.41 in the 
case when E' := E © Kv' is isotropic, i.e., when E' Q has a torsion free isotropic 
vector. The anisotropic case is simpler and is left to the reader. We first develop 
some generalities in the following two subsections. 

5.1. Commuting positive forms. Let F be a free K-module of finite rank. By a 
positive form on F, we mean a M-bilinear map 

[, ] F : F x F -> K 

satisfying 

[u, v]p = [v, u]f, [au, v]f = a[u, v]f, a £ K, u,v <E F. 

and 

[u, u]f £ for all torsion free u £ F. 

Here a £ K denotes the componentwise complex conjugation of a, for a £ K. 

Now further assume that F is a hermitian K-module, i.e., a non-degenerate her- 
mitian form (, )f (with respect to r) on F is given. We say that the positive form 
[, ]f is commuting (with respect to ( , )f) if 
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where Op : F — > F is the IR-linear map specified by 
(19) [u, v]p — (u, Opv)p, u,v E F. 

The following lemma is elementary. 

Lemma 5.1. Up to the action of\J(F), there exists a unique commuting positive 
form on F. 

Proof. We check the case of complex orthogonal groups, and leave other cases to the 
reader. So assume that (K, r) = (C, lc). Then 

[ , ]p t— > the eigenspace of Op of eigenvalue 1 

defines a U(F)-equivariant bijection: 

{ commuting positive forms on F } 

<-» { real forms F of F so that ( , )f\f xf is real valued and positive definite }. 

The assertion follows immediately. □ 

5.2. A Jacobian. Now fix a commuting positive form [, ]p, and denote by K(F) 
its stabilizer in U(F) (which is also the centralizer of Of in U(F)). Then K(F) is a 
maximal compact subgroup of U(F). Write 

F = F + ®F_, 

where F + and F_ are eigenspaces of Of of eigenvalues 1 and —1, respectively. 
With the preparation of the commuting positive forms, we set 

Sp := {u + v | u G F + , v G F_, [u, u]p = [v, v]p = 1, [u, v]p = 0}. 

Assume that F is isotropic, i.e., there is a torsion free vector of F which is isotropic 
with respect to ( , )p. This is the case of concern. Then Sf is nonempty. It is 
easy to check that K(F) acts transitively on Sf. According to [ShtJ, Sf is in fact a 
Nash-manifold. Furthermore, it is a Riemannian manifold with the restriction of the 
metric 

1 

triK/M | , \p. 



diniR IK 
Write 

I>-i := {u G F | (u,u) F = -1}, 
which is a Nash-manifold. It is also a pseudo- Riemannian manifold with the restric- 
tion of the metric 



dim R K 

Equip IR+ with the invariant Riemannian metric so that the tangent vector £4 
at t G has length 1. As a product of one or two copies of K+, is again a 
Riemannian manifold. 
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Define a map 



r] F : Sf x K y 4 



F,-l 



{w,t) h-> 



where 



w — u + v, w G F + , t> G F_ . 

Note that the domain and the range of the smooth map rjp have the same real 
dimension. Denote by J VF the Jacobian of rjp (with respect to the metrics defined 
above), which is a nonnegative continuous function on Sf x K+- Since 77^ and all the 
involved metrics are semialgebraic, J VF is also semialgebraic (see [Sht] for the notion 
of semialgebraic maps and Nash maps). Note that K(F) acts transitively on Sp (and 
trivially on K+), r] F and all the involved metrics are K(F)-equivariant. Therefore, 
there is a nonnegative continuous semialgebraic function Jp on K* such that 

J VF (w,t) = J F (t), w G Sf, t G &*. 

Denote C(X) the space of continuous functions on any (topological) space X. 

Lemma 5.2. For G C(r^_i) ; one /ias t/iat 

4>{x)dx — - I Jp(i) / <f)(r] F {w,t)) dw d x t, 



where dx, dw and d x t are the volume forms associated to the respective metrics. 
Proof. For every t G Ki, write 



(20) (t) := 

Write 



£, if d K = 1, 

tit 2 , if c? K = 2 and t = (ti, t 2 ). 



r fr i(i) = {«er frl | (K M ] F ) = i}, 

which is a closed submanifold of Tp-i of measure zero. One checks case by case that 
t)f induces diffeomorphisms from both 

S F x {t G K* I (t) > 1} and S F x {t G K* | (t) < 1} 

onto rV 5 _i \ r^_i(l). The lemma then follows. □ 

5.3. A preliminary integral formula on Gq\G' . We recall the notations of Sec- 
tion [2j The isotropic condition on E' = Eq © Kv' Q (which we assume as in the 
beginning of this section) ensures that there is a vector t> G E such that 

(V Q ,V ) = 1. 
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Denote by Z its orthogonal complement in E . Then E is an orthogonal sum of 
four submodules: 

(21) E = (X r © Y r ) © Z © Kv © Ku£. 

Fix a commuting positive form [, ] E on E so that (|2ip is an orthogonal sum of five 
submodules with respect to [, ] E . Recall that 

G:=XJ{E), G' :=XJ(E' ), G = XJ(E ). 

Put 

K:=K(E), K' :=K(E' ), K :=K(E ). 
For every t G K*, denote by g t G G' Q the element which is specified by 

{9t(v + v' ) =t(v + v' ), 
9t(v - ^o) = t~ T (v - v' ), and 
9t\x r ®Y r ®z = the identity map. 

We use the results of the last two subsections to prove the following lemma. 

Lemma 5.3. For (f) G C(Gq\G' ) , we have 

I (f)(x)dx= / J E > o (t) / <p(g t k)dkd x t, 
Jg \g' JkI Jk' 

where dk is the normalized haar measure on K' , and dx is a suitably normalized 
G' -invariant positive measure on Go\G f . 

Proof. By first integrating over K' Q , we just need to show that 

(23) / 4>{x)dx= [ J K {t)<f>(g t )d x t, <fr G C(G \G' /K' o ). 

Jg \g' Jr.* 

We identify G \G' Q with _i by the map g \— > g~ x v' . Note that v + v' G S E ^ 
and G g t is identified with r] E ^(v + t^t -1 ). The measure dx is identified with a 
constant C multiple of the metric measure dy on T E > _ 1 . 

Let 

<j ) eC(G \G' /K' ) = C(K' \T K _ 1 ). 
Then the function 4>(r] E ' o (w,t' 1 )) is independent of w G S E > Q . Also note that 

JE> {t) = J E > (t- 1 ), * eK*. 
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Therefore by Lemma [5.21 we have 



4>(x) dx = C / 4>(y) dy 

G \G' ^ t e',-i 



K? JS F , 



\c iJ E >(t) I <j>(r]E>(™,t))dwdH 

1 
2 

1 
2 



c / ^'(r 1 ) / ^(w.r^dwdH 



C J E <{t) <j>(71 K (v + V , ,t- 1 ))dwd*t 



= / J^mgt) [ IdwdH. 
1 Jk* Js f , 

We finish the proof by putting 



C := 2 I / 1 diu 



s P / 



□ 



5.4. The integral formula on S r \G. Denote by B r the Borel subgroup of GL r 
stabilizing the flag 

= X C X l C • • • C X r . 

For every t = (tx, t 2 , ■ ■ ■ , t r ) G (K*) r , denote by a t the element of GL r whose restric- 
tion to 

{veXi\ [v,x l _ 1 ] E = o} 

is the scalar multiplication by t{, for i — 1, 2, • • • , r. 
Proposition 5.4. For (j) G C(S r \G), one has that 

/ <f)(g)dg= <f)(atgtk)5p^{a t )5 B l{a t ) J E [ ) {t)d x td x tdk, 

Js r \G J (K*) r xKlxK 

where dg is a suitably normalized right G-invariant measure on S r \G. 
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Proof. Write K r = K D GL r . Then we have 



(f)(g) dg = <f)(g) dg 

S r \G J(N Pr N r G )\{N Pr GL r G' K) 



4>{rng' 'k)8 p 1 (m) dm dg' dk 

(N r \GL r )x(G \G' )xK 

4>{a t lg' 'fySp 1 (a t ) 8p 1 (a t ) d x t dl dg 1 dk 

(KlYxK r x(G \G' )xK 

(fi(atg'k)5p^(at) S^(a t ) d x t dg' dk 

K*y X (G \G' )xK 

(I G K r C GL r commutes with g' G G' ) 

0(fltfi , i^) ( ^p 1 (flt) <5s 1 ( a t) -Te' W d x t d x t dl dk 
(By Lemma [5. 3 P 

(f)(a t g t k)6pi(a t ) 5 B l r {a t ) J E /(t) rf x t rf x t dfc. 

(K^xKjx-R: 



□ 



6. Proof of Proposition 13.41 
6.1. An Iwasawa decomposition. Recall that we have a hermitian K-module 

(24) E' = X r © Y r ®Z ® Kv © Kv' © Kv'. 

Equip it with the commuting positive form [, which extends [, \e and makes v' 
and E perpendicular. Also recall that G' := U(E'). Put K' := K(E'). 
Write 

(25) E 3 = Kv © Kv' © Kv', 

and denote by Ne 3 the unipotent radical of the Borel subgroup of U(_E 3 ) stabilizing 
the line K(t>g +v'). For every t G K*, denote by b t G U(£ , 3 ) the element specified by 

(26) { b t {v' + v')=t{v' + v'), 



b t (v> -v>)=t-T(v' -v'). 
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For every t G {t G | tt T = 1}, denote by c t G U(_E 3 ) the element specified by 

c t (v ) = tv , 

C *K) = v' , 
ct(v') = v'. 

Recall the element g t G G' C G' in (1221) . Note that it also stays in XJ(E 3 ). By 
Iwasawa decomposition, we write 

(27) g t = c t „n t b t >k t , n t G N Ea , k t G K(E 3 ). 

Then both t' and £" are Nash functions of t. 
Lemma 6.1. One has that 

t' = 2(t~ 2 + t 2T + 2)"5. 

Proof. Note that Vq,v' ,v' is an orthonormal basis of E 3 with respect to [, ]&'■ We 
have that 

[9t\ v 'o + v'),gt l {v' + v')} E , 
= [fcf V™i~VK + v'), fc t " V"T VM> + v')] E , 

E' 



[t'-'iv'v + v'W-^ + v')} 
2t'~ 2 . 



On the other hand, 



1 / / f\ ^ / / 

9t ( v + v ) = n V ° + = V Q + V > 



+ 1 



and 



2 

t- 2 + t 2r + 2 
~ 2 ' 

Therefore the lemma follows. □ 
6.2. Majorization of Whittaker functions. We define a norm function on G' by 

:= max{(([gu } gu] E >})* \ ue E', ([u,u] E ') = 1}, 9 G G", 
where (•) is as in (|2"0|) . 



(28) £(t) 
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For every t = (t\, t 2 , ■ ■ ■ , t r , t r+ \) G (K+) r+1 , write 

m =1 (l + ^)' ifdK = l, 

m=i(i + ^) >< m=i(i + £y , a = 2 and *, = *,, 2 ). 

Write 

«t = Q t & tr+i G GL r +i, with t = (ti, t 2 , ■ ■ ■ , t r ). 
Recall that a t is defined in Section [5741 and b t is defined in fl26l) . 
Following |Jacj Proposition 3.1], we have 

Lemma 6.2. Let notations be as in Section lST^ . Let c p be a positive number, \-\ no a 
continuous seminorm on 7i , and |-| Pj0 a continuous seminorm on p. Assume that 

\ A (p(g) u )U < \\g\\ cp \ u \ P ,o, g &G x GL r+ i, ue p. 

Then for any positive integer N, there is a continuous seminorm \-\ p ,n on P such that 

\h{p{ai)u% < ^t)- N \\ai\\ e » \u\ p , N , t G (K*) r+1 , u G p. 

Proof. To ease the notation, we assume that dj& = 1. The other case is proved in the 
same way. For every i — 1, 2, • • ■ , r, let Yi be a vector in the Lie algebra of GL r+ i so 
that 

Ad a _Yj = - — Yi, t = (t\, t 2 , ■ ■ ■ , t r+ x), 

H+l 

and 

rtii := -ip r+ i(Yi) ^ 0. 

Here if> r +i stands for the differential of the same named character. Similar notations 
will be used for the differentials of representations. 

For every sequence N = (N±, N 2 , ■ ■ ■ N r ) of non- negative integers, write 



t (N) := H(ti/t i+1 ) Ni , t= (h,t 2 ,--- ,t r+1 ) G (K^j 



i=l 

Also write 



yN _ y^y 2 2 ■ ■ ■ Y Nr 

which is an element in the universal enveloping algebra of the Lie algebra of GL r+ i. 
Then 

A(p(a t )p(y» = A(p(Ad ai Y N )p(a- t )u) 
= t( N )A(p(r>(a^) 

flmAi^Aip^u). 

J=i J 
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Therefore 

(29) t( N )|A( P (a»u < H- N |k|Mp(r>u , 

where 

r 

i m |-N := Y[\ mi \- N >. 

i=l 

Given the positive integer N, write 

e(tr = ^a N tw, 

N 

where on's are nonnegative integers. In view of ( 1291) . we finish the proof by setting 

\u\ P} n ■= ^ a N |^r N |p(y N )M|p,o- 

N 

□ 

6.3. Convergence of an integral. 

Lemma 6.3. For any non-negative continuous semialgebraic function J on (K*) r+1 ; 
there is a positive number cj with the following property: for every s G IR dK with 
s > cj, there is a positive integer N such that 

(30) / {hh t r t' r+1 ) S £(t u t 2 ,--- ,t r X+l)~ N J(t)d X t < 00, 

7(K^)'-+ 1 

where 

t = (ti, t 2 , ■ •• , t r , t r+ i), t' r+1 = 2(t r + 1 + t^+i + 2) 2 , 
and £ is defined in (1281) . 



Proof. To ease the notation, we again assume that c/k = 1. Note that the change of 
variable 

7 ~ f tl ^2 , 

t^a:= I Qfi = —, Qf 2 = —,••• , a r _i = — — , a r = - — , t r+1 

is a measure preserving Nash isomorphism from (K+) r+1 onto itself. So J is also 
a continuous semialgebraic function of a. It is well known that every continuous 
semialgebraic function (on a closed semialgebraic subset of a finite dimensional real 
vector space) is of polynomial growth (|BCR, Proposition 2.6.2]). (This is the reason 
that we work in the semialgebraic setting in this article.) Therefore there is a positive 
number c'j such that 



x\r+l 
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Take a positive number cj, large enough so that 



I t' r+1 {r+1)cj (t r+1 + 1;^ dH r+l < oo. 



and 



/ af J (aj + a j l ) c J d x aj < oo, j = 1, 2, 
The integral f[3"Uj) is equal to 

/ . . . c-^+Y^ TT(i + «i) _iV J («) rf 



^(n/ " f "w" 7 x / < +1 (r+1)s (Wi + ^) c ^ x Ui- 



K j=i 



[1 + aj) 



Now it is clear that the above integral converges when s > cj and N is large 
enough so that 

□ 

6.4. End of proof of Proposition [3741 Take a continuous seminorm j-l^o on 7r 
and a continuous seminorm | - | Tro on 7r such that 

(31) \fi(u,v)\ < \u\ nfi ■ \v\ no , u e 71, V E 7l . 

Take a positive integer c n and a continuous seminorm \-\ n ^ on 7r such that 

(32) \ir(g)u\n i0 < I^H^H^i, g E G, u E n. 

Take a positive integer c p and a continuous seminorm |-| Pi o on p such that 

(33) \A(p(g)(u))\ V0 < \\g\\ Cp \u\ pfi , g E G x GL r+1 , u e p. 
Now assume that 

g = a t g t k, t = {t u t 2 , ■ ■ ■ , t r ) E (I^) r , tE^kEK. 

Write 

9t = c t nn t b t >k t 
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as in f|27|) . Then for all s G C dK , u G ir, f G tt' s , we have 

li{iT(g)u, A(f(g))) = 11(11: (ct»n t b t/ a t ktk)u, A(f(c t nn t b tl a t k t k))) 

{k t G 1J(E 3 ) commutes with a t G GL r ) 

= fi(n(n t b t ,a t k t k)u, A(f(n t b t >a t k t k))) 

(By Lemma [3.21 and the fact that Cf G Go C S r ) 

= fi(n(n t b t >a t k t k)u, (tit 2 ■ ■ ■t r t') s A(p(a t bf)f(k t k))) 
(n t G Np> r+1 and a t b t > G GL r+1 ). 

Therefore by (13T1) . (1321) and the fact that the norm function || • || on G' is right 
i^'-invariant, we have 

(34) \p(rc(g)u, A(f(g)))\ 

< (t!t 2 ---M') Re(s) x IhMtir x K,i x |A(p(at&tO/(M0)L- 

Let J be the (nonnegative continuous semialgebraic) function on (K^) r+1 defined 

by 

J(t,t) := Ih^tir x ||a t ^|| Cp x 5p r X (a t ) (J^(at) 



Let c M := cj be as in Lemma [6.31 and assume that the real part Re(s) > c M . Let N 
be a large integer as in Lemma 16.31 so that 

c s , N : = [ (t 1 t 2 ---t r t') Rc(s) ^(t,t')- N J(t,t)d x td x t<oo. 

Take a continuous seminorm \-\ Pt N 011 P as i n Lemma [6.21 Then we have 

(35) \A(p(a t b tl )f(k t k))U < C(t,t')- N \\a t b tl \\ c "\f(k t k)\ P!N < £(t, t') _JV | k&tf I M/U 
where 

|/|^ :=max{|/(A;')| P ,v|A;'Gir'}, 
which defines a continuous seminorm on 7r£. Combining (I34|) and (|35|) . we get 

(36) |/i(7r(<7KA(/(<7)))| 

< x l/l^ x (M 2 ---M') Re(s) x ^(t,0 _iV x |h^a t |r x ||a t ^|| Cp . 
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Then by Proposition 15.41 and (|36|) . we have 
/ \fi(-K(g)u,A(f(g)))\dg 

JSr\G 

= / \fj,(n(atg t k)u, A(f (a t g t k)))\ 5p}(a t ) 5 B l(a t ) J E[) {t) d x t d x t dk 

J(K*) r xK*xK 

< |u| w ,i x \ f\ n , x / (M 2 • • ■ M') Re(s) £(t, t')- N J{t, t) d x t d x t 

= C S , N X l^l^i X \f\ w f. 

Therefore the integral J M (/, it) converges absolutely. Finally, 

|/^(/,m)|< / \p{ir{g)u, A(f(g)))\dg< c S)JV x |u| ffil x 1/^, 

•/S r \G 

which proves the continuity of J /t . This finishes the proof of Proposition 13.41 
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